A Comparison of Approaches for Maximizing Business Payoff of Prediction Models

In many database marketing applications the goal is to predict the customer behavior based on their previous actions. A usual approach is to develop models which maximize accuracy on the training and test sets and then apply these models on the unseen data. We show that in order to maximize business payoffs, accuracy optimization is insufficient by itself, and explore different strategies to take the customer value into account. We propose a framework for comparing payoffs of different models and use it to compare a number of different approaches for selecting the most valuable subset of customers. For the two datasets that we consider, we find that explicit use of value information during the training process and stratified modelling based on value both perform better than post processing strategies. 4% base rate), giving the model a Zifc of 30/4-7.5. For a large customer base, even small improvements in prediction accuracy can yield large improvements in lift. In this paper we argue that lift measure by itself is not sufficient and that we should take the customer value into account in order to determine the model payoff. Using the predicted behavior and a simple business model we estimate the payoffs from different models and examine different strategies to arrive at an optimal model that maximizes overall business value rather than just accuracy or lift. In the rest of this paper we explain the business problem and model of business payoffs, present the main experimental hypotheses, results and conclusions.